Until recently, image processing systems have generally processed images, such as frames of video, still photographs, and the like in an individual manner. After processing, the individual images are combined to form a mosaic, i.e., an image that contains a plurality of individual images. Additional image processing is performed on the mosaic to ensure that the seams between the images are invisible such that the mosaic looks like a single large image. The alignment was previously done manually by a technician to remove the seams. In such computer aided processing systems, the technician manually selects processed images, manually aligns those images, and a computer applies various images combining processes to the images to remove any seams or gaps between the images.
More recently, systems for automatically aligning images to form a mosaic image have started to gain acceptance. One such system is disclosed in U.S. Pat. No. 5,999,662. In the '662 patent, the image alignment process automatically aligns one input image to another input image, an input image to an existing mosaic (generated from previously occurring input images) such that the input image can be added to the mosaic, or an existing mosaic to an input image. In each of these instances, the coordinate system of the aligned images is either the coordinate system of the input image, the coordinate system of the mosaic, or an arbitrary reference coordinate system. The input image in mosaic can be aligned to one another within an image pyramid frame work. As such the system converts both the input image and the mosaic into Laplacian image pyramids in the alignment processes applied to levels within the respective pyramids. Consequently, the system uses a course-to-line image alignment approach that results in sub-pixel alignment accuracy. This process generates a pyramid of continuing lower resolution images. Interative registration of lower and higher resolution images yields final product.
The difficulty with the prior art can best be explained by referring to the zoom capabilities of the camera. The camera can zoom in or out. As the zooming occurs, a structure within a frame will change position relative to the image. For example, if the image has a tree and the tree is in the center of the image as the camera zooms in the camera is going to get apparently larger within the frame. As the camera zooms out the tree will become apparently smaller. The distances from the center will change the function of how far the camera is zoomed in or zoomed out. This presents a problem with the prior art in trying to register from frame to frame. The sequence of pixels in one frame is going to be a difference sequence of pixels in another frame. Further, the prior art system disadvantageously requires special hardware.